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(54) Visual attention location system 

(57) This invention identifies the most significant 
features in visual scenes, without prior training, by 
measuring the difficulty in finding similarities between 
neighbourhoods in the scene. Pixels in an area that is 
similar to much of the rest of the scene score low meas- 
ures of visual attention. On the other hand a region that 
possesses many dissimilarities with other parts of the 
image will attract a high measure of visual attention. 
The invention makes use of a trial and error process 
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to find dissimilarities between parts of the image and 
does not require prior knowledge of the nature of the 
anomalies that may be present. The method avoids the 
use of processing dependencies between pixels and is 
capable of a straightforward parallel implementation for 
each pixel. 

The invention is of wide application in searching for 
anomalous patterns in health screening, quality control 
processes and in analysis of visual ergonomics for as- 
sessing the visibility of signs and advertisements. 
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quence. inVen "' 0n rS,ateS l ° 3 SyStSm ' 0Catin9 Sa ' ient within a static image or a video se- 

or surrounding population without the orfor Sfn nn 1 ' h S ° me reSpeCt fr0m the background 

applications lerL^ZTZlZ Te^S^^T"™" "°™ er - there 

very large. A particular example is th™n£eSo^mL^ m ^ rSPetitiVe a " d the V0,ume of data * 

' such situations, where a large number of samples hav ^o tT f?"!? ^ * iden,ify Cancerous ce,ls - 
become inattentive, J overlooMh^e^ "* ^ 3 h ™ 

in - L ma9e W0UW a,S ° bS — - -V other 
of information signs and as a suSut ^ oMen Q ,hv and 1x1^ ^ 9 V ' SUal SC6ne ' the d6Sign and locati °" 
10004] ^tingsystemsforgaugingv^^ 

visual matL, t^Jii^S^SSZZ TsZtTs^ T^o ^ "° ^ ° f hand,ina new 
be too anomalous to be identified as such bv « ? o SyStem ' Parado *i ca »y, a feature may simply 

resource in order to process he pr 111^ ^1^ ^ T" ^ Considerable computational 

of the procedure is e'xtended ar '^^S^^^^ *" """^ ,ne ""~ With ° Ut » mit as the ~°P* 

storing an image as an array of pixels, each pixel having an intensity value- 
selecting test pixels, y ' 

an anomaly ,al„. is i„o"om.nie7,ri. , nron»« ' , ? C °™P°"«'»9 PMI neighbouring g,. comparison ptal, 

same test pixel 

correspond^ pix^i^ l S^fl2 r C^ x : , ^1^,^ ^ ™ ^ Mnti ^ simi,ar 

as the measure of visual attention for that Sxel an ° ma ' y Va ' Ue ^ eaCh ,eSt piX6 ' is 0Ut P ut 

Seltharma 0 ,;^ 

includes the Jteps ^Ztog?J^^ u £^" of subsequent test pixels. Preferably, therefore, the process 
selecting, for subsequent Test ^a n^^JT'™ '? ^ an ° ma ' y ValtJe has been Benerated, and 

subsequent test pixel as JT.^V^u^^ST' " ^ P ° Siti ° nal re ' ati ° nShipS <° the 

ScTsS^ a 9 eo ra 

means for storing an image as an array of pixels, each pixel having an intensity value- 
means for selecting test pixels, y ! 
means for selecting neighbour sequences of pixels neighbouring the test pixel- 
means for selecting comparison pixels having intensity values substantially the same as the t«t 

™.n, ,., o=n e „„„o , mmu „ o, v,su., ,n.n„on to, each »„ p,„,. o.ing m . „„ mBer o( oomp , rtsom mMe 
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that test pixel before a matching sequence is identified. 

Sn«l l h>S apparatus is P refera bly embodied as a general purpose computer, suitably progarmmed 
[0010] The invention also extends to a computer programmed to perform the method of the invention and to a 
computer program product directly loadable into the internal memory of a digital computer, comprising software code 
portions for performing the steps specified above. 

[001 1 ] According to another aspect, the invention provides a computer program product stored on a computer usable 
medium, comprising: M 
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computer-readable program means for causing a computer to store an image as an array of pixels each pixel 
having an intensity value; ^ 
computer readable program means for causing the computer to select test pixels 

computer readable program means for causing the computer to select, for each test pixel, neighbour sequences 
of pixels neighbouring the test pixel; 

computer readable program means for causing the computer to select comparison pixels having intensity values 
substantially the same as the test pixel; 

computer readable program means for causing the computer to identify the sequence of pixels neighbouring a 
selected comparison pixel having the same respective positional relationships to the comparison pixel as a selected 
neighbour sequence of pixels has to the test pixel; 

computer readable program means for causing the computer to compare the intensity values of the selected neigh- 
bour sequence with the intensity values of the corresponding pixels neighbouring the comparison pixel 
computer readable program means for causing the computer to generate a measure of visual attention for each 
test pixel, being the number of comparisons made for that test pixel before a matching sequence is identified. 

[0012] The invention may be used to identify the principal subject in a visual scene, by identification of the region 
containing pixels having the greatest anomaly values. It may be used to determine a measure of visual attention afforded 
to a given object in a visual scene by comparison of the anomaly values generated for the pixels representing that 
object with the anomaly values generated for other parts of the scene. 

[0013] The repetitive nature of the process lends itself to parallel processing, and it should be understood that several 
test pixels may be processed in parallel with each other, and for each test pixel, several sequences of neighbour pixels 
may also be processed in parallel. H 
[0014] In order to allow for minor variations in intensity between otherwise similar elements of a scene the compar- 
ison of intensity values preferably allows a small difference in intensity values between two pixels to be considered a 
match, and the term "substantially similar" used above should be understood in that context. The value of this threshold 
difference may be varied for different cycles, those values which produce a suitable distinction between elements being 
stored and re-used on subsequent cycles of the process. 

[0015] For a colour image the intensity values are three-element (red, green, blue) vectors 

[0016] This invention identifies saliency in visual scenes by measuring the difficulty in finding similarities between 
neighbourhoods in the scene. Pixels in an area that is similar to much of the rest of the scene therefore score low 
measures of visual attention, so are not considered to be worthy of attention . On the other hand a region that possesses 
many dissimilarities with otherparts of the image will attract a high measure of visual attention, as the anomaly values 
scored will be large. 

[0017] The invention makes use of a trial and error process to find dissimilarities between parts of the image and 
does not require prior knowledge of the nature of the anomalies to determine saliency. The method avoids the use of 
mn^f a 8 de P 6ndencies between P jxels and is ca Pa°le of a straightforward parallel implementation for each pixel 
[0018] A preferred embodiment will now be described, by way of example, with reference to the figures, in which 

Figure 1 illustrates the process schematically; 

Figure 2a represents an image to be processed by a method according to the invention, illustrating the comparison 
process for two sets of pixels Xj, y f 

Figure 3a represents a second image to be processed by a method according to the invention 
Figures 2b and 3b are mappings of the anomaly values generated for these images 

Figure 4 illustrates schematically the basic components of a general purpose computer capable of performina the 

invention. w 

l °V?L, V? com P° nents ^strated in Figure 4 comprise an input means 41 , such as a scanner, a central processing 
unit (CPU) 42, an output unit such as a visual display unit (VDU) or printer 43, a memory 44. and a calculation processor 
45. The memory includes stores 440, 444 - 446, registers 441 , 447 - 449 and counters 442, 443 The data and the 
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\n^l^u S ' Ma Fi9UreS 1 4 ' thS ima9e 40 t0 be analysed is accessed b V th * '"Put means 41 and stored 
in a digital form in an .mage store 440, as an array A of pixels x where each pixel has colour intend , „ 
* attnbuted to it, in the case of grey level images, a single grey scale intensity value L ^ 9 *' ^ 

[0021] A pixel x is then selected from the array A (step 1 ) and its intensity value it « h\~> ♦ - • 
S ster44,Sever al ^ 

» ss >*:^^^:^~:^ nd a count of the — ° f «- ~ ^ 

E^L* f & T T te9V iS th6n S6 ' eCted by the CPU 42 (ste P s 3 . 4 . 5) and provided to a neighbour group definition 
store 444. Each such strategy comprises a set of colour difference thresholds (Ar x , Ag x , Ab,) ( 0r ,„ Z case o ^ ev 
level images a sing.e threshold Ay, (step 3) and a neighbour group definition (s eps 4,5 I nitial y the ^search sLtea es 

» ^SSfbS^S a n 1 " 6 Tl " ! ^ S,rate9y ^ SUit3ble f ° ! identifyi " 9 ^ere^s r c Xil b 

^.^i^^ ected ' successfui strate9ies can be stored in a search s,rategy st ° re 

Th°^LI he C ?^ Ur K if !! renCe thresh0,ds selected in ste P 3 determine whether two pixels are to be considered similar 
The d fference thresholds must exceed a certain minimum otherwise no similarities will be detected, but if Tey are too 
great too many similarities will be found. y 

20 5Sf 51 • T ° t define 9 neignboiJr 9 rou P a radius u * ^ selected at random within certain bounds (step 4) This value 
P r^ 

^n/nn nnl , u x m determ,ned b V ,he scale of the features that establish visual attention which will 

S , C T P ° S . e Wh ' Ch ima9e iS 10 66 analyS6d - AS With the diffe ™ ce thresholds, the selection I random 
withm these limits, selections which fail to provide a distinction being rejected (step 9) 

S 0 A 6 oL A t h eqU T C f ° f " PiX6 I S Xj jn nei 9 hboum °° d of the test pixel x is selected from the image store 440 (step 
5). Again, this selection is random, the selection being such that: 
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dist(x,, x a .„) < u x 

where j = 1, n and x 0 = x 

ES, ,?J X r Pl ° °' f"f a „ S eqUence is shown in Fi 9 ure 2. in which the test pixel (shown boxed) has a sequence 
(shown shaded) assorted with ,t. Typically n = 3 , and u x = 1 . In some cases u x may vary with j- this allows lels tl 

oi 'oST, x 7™ 2 rr" 5 ' enSUrin9 that S ° me ° f ,he Se,6C,ed piX6ls are c,ose 2 the test p"e x^The Sue 

stored n\ e ° nLhh' h r ^ SU ' tab ' e Unit8 ' SUCh 33 piXSl SiZe ' The definition of the sequence is 

stored in the neighbour group definition store 444. 

[0028] Previously generated search strategies, comprising neighbour pixel sequences definitions x- and associated 
colour difference thresholds (Ar x , Ag x , Ab x ) stored in the search strategy store 445 as a resuS of achievtnT 5 
anomajfscoreon previous test pixels (step 11 , to be discussed) may bep«i^mia.^ 8 electedbythrcw« 
generated candidates only being supplied by the processor 42 tothecurrentnrtghbourgroupdS^^^ 
orocelT ?„? "? ^1°^ is exhausted - ™» ^chanism reduces the number of unsucces fu i eraL^s oTthe 

[0029] Similarly, when processing many similar images (for example in a moving image, or any other larae set of 

[0030] Pixels y are identified as being similar to the test pixel x if: 

IV r x| <Ar x. |g y -g x |<Ag x , and |b y -b x |<Ab x . 
[0031] For grey level images Ity - t x l < At x . 

stored in S im^f 7 '"IT Ca ' CUlati0n processor 45 ' which the intensity vaiues of image pixels 

TrLlt h > 9 6 440 W ' ,h thS ' ntensity Va,ue of the test P |xel stored in the test pixel register 441 using the 
threshold criteria Ar x , Ag x , and Ab x stored in the current neighbour group definition store 444 The pixels thi match 
withm the limits defined by the threshold criteria, are stored in a comparison pixel store 446 and S v arT then 

^S^tSr St0rS ^ 6) t0 bS CUrrSnt " C ° mPariSOn PiXel " (a ' S ° ^own boTed in y FiS e 2 
whose identity » stored ,n a comparison pixel register 447. If no suitable pixel can be found after a certain number of 
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SS ,t8 '^ h e " T th ^ Sh °' d ' eVel iS rSdefined (StSP 3) ' and the P rocess restarted f ™ that point 
[0033] The value of l x stored in the comparison counter 443 is incremented (step 7) The contents of the nPinhho„ r 
group definition register 444 are then used by the calculation processor 45 to define a ieto S £™ n „! f ne ' t ghb0ur 
x, (register 448) and a set of pixels forming a comparison group y, (register ^sfeLh oixe v ofthlT 9 9TOUP 

for ™ll ,t ?„ PP ' y 3 " eW com P arison neighbour group to the comparison group register 449 

whe theThresho.d slashed hL th " " PiXe ' * the '° Wer tne va,ue ° f the an °™'V value 

wnen ine tnresnoia L is reached by the comparison counter 443 

in° rt 3 nL h A h S Pr0C6SS COntinues < su «*ssful search criteria (that is. combinations of values of Ar Aq Ab and u 
•3 th *~7^ U T^T CU 9Snerate hi9h ValU8S ° f ^ Wi " beCOme a PP a ™'- " a -q-nce ofn'pfxis x and 

by retrieval from the search strategy store 445 or generated randomly. P ' '* 

™L r* 8 "/ ^ « rati ° n Va ' Ue '* St0rSd in the com P arison ~"nter 443 reaches a threshold value L the iterative 

from those p.xels having the same inherent characteristics as pixel x s y ranaomly selected 

eel 4 , •&^ , 2£srir~ a ' ,or *~ - M '" "•• " K - ,h - — - * 

[0041] T neprocesspossessesseveraladvantagesoverotherprocedures.Firstlytheprocessmak e sn Qass „ m ntinn 0 
about the content of the image and is able to extract useful features relevant to Xto^Z^™^ r^™Z 
process and hence is able to adapt to the materia, in any image. Secondly the process appliL eoua^.v Z anv Z T 
uration of pixels whether arranged in a rectangular W .a.pian«y 1 ori fc ^^S"K1S^ 
be app„ed to each p.xel x: without any dependency on the computations associated with other pi and hen^ 2% 
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be applied in parallel to many pixels simultaneously. This means that with a parallel implementation results may be 

fh>K T T T° mat6ri !' iP rea ' tifTle ' ° r 6Ven f3Ster Fourthly the al 9° rithm is based u P° n an evolutionary procedure 
which has the advantage that trials do not have to be prepared with the rigour normally afforded software processes 
Some cycles may not produce useful results, for example because they contain obvious redundancy (eg a sequence 
of neighbour p.xels Xj wh,ch includes the same pixel more than once). Such cycles are rejected in the same way as 
any other cycle that fails to identify distinguishing features, without any special rejection process being necessary to 
identify such sequences. This effectively removes the computational burden required to accurately construct viable 
candidates for trial. 

[0042] In the following simplified examples, the process has been applied to black and white images consisting 
entirely of ones and zeros. In this case Al t = Jft. n = 3, L = 100, and u, = 1. The first example (Figure 2a Figure 2b? 
exemplifies the classical problem of "popout" in which certain types of shape stand out if they are surrounded by 

different shapes. * 

[0043] The measures of visual attention attributed to each pixel in Figure 2a are shown in the chart in Figure 2b The 
vertical scale indicates the anomaly value (expressed as a percentage of the number of attempts L) for each pixel It 

S l en anomaly values C| are very much higher for the diagonal bar than for the vertical bars 

[0044] F,gure 3a illustrates the effect of clustering on visual attention where a grouping of vertical lines is placed 
amongst others that are more widely separated. The results of using the process of this invention are shown in Figure 
3b. Again, the clustered lines produce a higher anomaly score. 

[0045] Jt will be noted that the process does not require any previous knowledge of the nature of the anomalies being 
searched for. The anomaly may be in orientation (as in Figure 2a), spacing (as in Figure 3a), shape, length colour or 
any other characteristic. 

[0046] The invention is of wide application in a number of fields. Firstly, identification of the principal subject in a 
visual scene is the first essential stage in the categorisation of unfiltered visual content - it is also the most difficult 
Once this step has been achieved, it may be followed by manual tagging, or a range of template matching or other 
automatic techniques for recognition of the features so identified. 

[0047] The ergonomic considerations in the design and location of warning signs (eg road signs) in order to render 
them conspicuous is often a process of trial and error, with risk to the public during this phase. An objective measure 
of visual attention (in other words, identifying whether the sign, or something else, is the principal subject in the intended 
audiences view of the sign in its proposed environment) would improve the design process and reduce the risk of 
accidents caused by insufficiently conspicuous experimental signs. The visual impact of other signs, such as adver- 
tisements, and the layout of display screens such as Internet "Websites", can also be optimised using this process to 
maximise visual attention in specific locations. 

[0048] Secondly, the invention would be capable of identifying objects that are different in some respect from the 
background or surrounding population. For example, a cancerous cell may be identified by its possession of features 
not present in the surrounding healthy cells. The identification of such anomalies by eye is currently a very labour- 
intensive process, because of the large number of samples to be inspected and the comparative rarity of cancerous 
r«n!„, T an observers have been known t0 fail t0 n °tice anomalous cells as a result of eye strain and fatigue 
[0049] As another example, in bubblechamberphotographs used by particle physicists, new and anomalous patterns 
of particle tracks may be identified by this process. Since the tracks of most interest are those generated by hitherto 
undiscovered particles with unknown properties, it is not possible to devise a template to search for them 
[0050] As a further example, the objective detection of defects in visual representations of textures would improve 
quality assurance of manufacturing processes of fabrics, microchip layouts and other processes where surface defects 
are to be avoided. 

l?°VL T hirdly ' recognition of the P^sence of objects that do not match their surroundings has many applications in 
the field of security surveillance. Such objects may constitute a serious hazard if not brought to the early attention of 
security personnel. Similarly, anomalous objects present in satellite images may reveal valuable intelligence information 
or local changes in the ecology. 

[0052] Finally, the invention may serve as a model of human visual perception with application to a range of tasks 
in wh,ch ,t is required to emulate human performance as a substitute for lengthy and expensive human factors trials. 



Claims 



1 . A method of processing a visual image or a sequence of such images, for locating areas of visual attention com- 
prising the steps of: 

storing an image as an array of pixels, each pixel having an intensity value; 
selecting test pixels, 
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for each test pixel, selecting one or more neighbour sequences of pixels neighbouring the test pixel- 
selecting comparison pixels having intensity values substantially the same as the test pixel- 
identifying the sequence of pixels neighbouring a selected comparison pixel having the same respective po- 
sitional relationships to the comparison pixel as a selected neighbour sequence of pixels has to the test pixel- 
comparing the intensity values of the selected neighbour sequence with the intensity values of the correspond- 
ing pixels neighbouring the comparison pixel, 

generating a measure of visual attention for each test pixel, being the number of comparisons made for that 
test pixel before a matching sequence is identified, 

A method according to claim 1 wherein, for each comparison pixel, if one or more of the selected pixels neighbouring 
the test pixel has an intensity value not substantially similar to the corresponding pixel neighbouring the comparison 
pixel, an anomaly value is incremented, and the process is repeated using further comparison pixels with the same 
test pixel until a comparison pixel is selected for which all the selected pixels have an intensity value substantially 
similar to the corresponding pixel neighbouring the test pixel, and the final value of the anomaly value for each 
test pixel is output as the measure of visual attention for that pixel. 

3. A method according to claim 1 or 2, wherein a plurality of test pixels are analysed concurrently. 

4. A method according to claim 1 , 2 or3 wherein a plurality of comparison pixels are compared with a given test pixel 
20 concurrently. r 

5. A method according to claim 1 , 2, 3 or 4, wherein the intensity value is a three-element vector representative of a 
colour image 

25 6. A method according to claim 1 , 2, 3, 4 or 5 wherein in addition to neighbour sequences, further variable search 
criteria are selected. 

A method according to claim 6, wherein the further variable search criteria include a threshold value for the deter- 
mination of whether two pixel intensity values are substantially similar. 

.A method according to claim 1 , 2, 3, 4, 5, 6 or 7, the method including the step of storing values for search criteria 
for which a high anomaly value has been generated, and selecting, for subsequent test pixels, the same search 
criteria 
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A method according to claim 1,2,3, 4, 5, 6, 7 or 8, wherein the principal subject in a visual scene is identified by 
identification of the region containing pixels having the greatest anomaly values 

1 0. A method according to claim 1 , 2, 3 : 4, 5, 6, 7 or 8 r wherein a measure of visual attention afforded to a given object 
in a visual scene is determined by comparison of the anomaly values generated for the pixels representing that 
object with the anomaly values generated for other parts of the scene. 

11. Apparatus for processing a visual image or a sequence of such images, for locating areas of visual attention 
comprising 

means for storing an image as an array of pixels, each pixel having an intensity value; 
means for selecting test pixels, 

means for selecting neighbour sequences of pixels neighbouring the test pixel; 

means for selecting comparison pixels having intensity values substantially the same as the test pixel; 
means for identifying the sequence of pixels neighbouring a selected comparison pixel having the same re- 
spective positional relationships to the comparison pixel as a selected neighbour sequence of pixels has to 
the test pixel; 

means for comparing the intensity values of the selected neighbour sequence with the intensity values of the 
corresponding pixels neighbouring the comparison pixel, 

means for generating a measure of visual attention for each test pixel, being the number of comparisons made 
for that test pixel before a matching sequence is identified. 

12. A computer programmed to perform the method of any of claims 1 to 10. 
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A computer program product directly loadable into the internal memory of a digital computer comD ris ina »°ftwi>r» 
code P ort,ons for performing the steps of any of claims 1 to 10 whenlaid product is run on a co^S. 

A computer program product stored on a computer usable medium, comprising: 

ZZZ !£2J ST means for causin9 a computer t0 store an imase as an ^ ° f ~* p^- 

computer readable program means for causing the computer to select test pixels 

^Z^SX^Z^Z 1 ^ 9 ,he computer 10 se,ect comparison pixe,s havin9 intensity 

r s T C te d r L a m able h meanS f ° r CaUSing COm P uter t0 identif y "quence of pixels neighbouring 

J^tT COm f anSOn P ' Xel hav,n 9 ,he same res Pective positional relationships to the comparison pixel as a 
selected neighbour sequence of pixels has to the test pixel- 

computer readable program means for causing the computer to compare the intensity values of the selected 

coTZ' S T h ? CS W ' th int6nSity Va ' UeS ° f th6 corres P°^'"g P** neighbouring the compar^on pSJ 
compute readable program means for causing the computer to generate a measure of visual attention fo 

fdentif ed " ' 9 ° f COmparisons made <°< tha < « pixel before a matching sequence is 
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For each pixel x 



Counts c and I 
set to zero 



Colour difference thresholds 
<r, g, b) set 



Neighbourhood 
radius u set 



Select n pixels in 
the neighbourhood of x 



Select random pixel y within 
colour difference thresholds of x. 



Figure 1 
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Figure 2a 




Figure 2b 
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Figure 3a 
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Figure 3b 
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